Measuring Machine Intelligence Through Visual Question Answering

Scenes The visual question-answering task requires a variety of skills. The machine must be able to understand the image, interpret the question, and reason about the answer. For many researchers exploring AI, they may not be interested in exploring the low-level tasks involved with perception and computer vision. Many of the questions may even be impossible to solve given the current capabilities of state-of-the-art computer vision algorithms. For instance the question “How many cellphones are in the image?” may not be answerable if the computer vision algorithms cannot accurately detect cellphones. In fact, even for state-of-the-art algorithms many objects are difficult to detect, especially small objects (Lin et al. 2014). Articles SPRING 2016 69 To enable multiple avenues for researching VQA, we introduce abstract scenes into the data set (Antol, Zitnick, and Parikh 2014; Zitnick and Parikh 2013; Zitnick, Parikh, and Vanderwende 2013; Zitnick, Vedantam, and Parikh 2015). Abstract scenes or cartoon images are created from sets of clip art, figure 7. The scenes are created by human subjects using a graphical user interface that allows them to arrange a wide variety of objects. For clip art depicting humans, their poses and expression may also be changed. Using the interface, a wide variety of scenes can be created including ordinary scenes, scary scenes, or funny scenes. Since the type of clip art and its properties are exactly known, the problem of recognizing objects and their attributes is greatly simplified. This provides researchers an opportunity to study more directly the problems of question understanding and answering. Once computer vision algorithms catch up, perhaps some of the techniques developed for abstract scenes can be applied to real images. The abstract scenes may be useful for a variety of other tasks as well, such as learning commonsense knowledge (Zitnick, Parikh, and Vanderwende 2013; Antol, Zitnick, and Parikh 2014; Chen, Shrivastava, and Gupta 2013; Divvala, Farhadi, and Guestrin 2014; Vedantam et al. 2015).